Systems and methods for dynamic digital signage based on measured customer behaviors through video analytics

ABSTRACT

A dynamic digital signage system based on measured customer behaviors through video analytics.

TECHNICAL FIELD

The presently disclosed embodiments are directed to signage systems,more particularly to a digital signage system, and still moreparticularly to a dynamic digital signage system based on measuredcustomer behaviors through video analytics.

BACKGROUND

Many retailers are transitioning from basic, static signage to moreengaging digital signage systems to attract customers. Existing types ofdigital signage can range from a simple design, to a more complicated,elaborate design which can be changed dynamically. Generally, retailersuse digital signage to update item prices without having to change papertags on shelves. However, this use of digital signage typically does notjustify the added cost of the digital signage. Similarly, retailers canuse digital signage to change larger promotional signage periodically(e.g., low frequency updates, generally static promotions, etc.). Thisuse of digital signage systems only presents information to customersfor a pre-determined length of time, such as, for example, during a saleevent, etc. Thus, there remains a need for an improved digital signagesystem which can be changed dynamically, and which can be changed basedon how engaging the signage is for customers (i.e. how much the signageholds the customers' attention).

More recent digital signage systems can include identifying features andor characteristics of an individual (e.g., age, race, gender, sex,height, etc.) to determine what advertisements and/or messages todisplay on the digital signage. Other recent digital signage systems caninclude identifying when an individual is actively viewing the displayto measure customers' attention. In such systems, the signage istypically switched when the customers are no longer engaged. As such, itwould be desirable to provide a dynamic digital signage system which canbe changed based on who is viewing the signage, and/or how engaging thesignage is to the viewer.

There is thus a need for an improved dynamic digital signage systemwhich can measure broader customer behaviors within a retailenvironment, and leverage the measured customer behavioral informationto adjust the information displayed on the digital signage.

BRIEF DESCRIPTION

According to aspects illustrated herein, there is provided a systemcapable of providing dynamic adjustment of digital signage in a retailstore based on video-based measurements of shopper state and behavior inthe surrounding scene, the system comprising: a video capture modulehaving at least one camera capable of capturing frame images of ashopper within the retail store; a video analytics module capable ofextracting information about the shopper's behavior and state from thecaptured frame images including selection recognition of shopperactivity relative to purchasable items comprising predetermined behaviorof physical handling of the items; a planner module capable of selectinga type of information to display on the digital signage based on theextracted information from the video analytics module; and, digitalsignage capable of presenting the selected type of information from theplanner module to surrounding shoppers in the retail store.

According to aspects illustrated herein, there is also provided a systemcapable of providing dynamic adjustment of digital signage in a retailstore based on video-based measurements of shopper state and behavior,the system comprising: a video capture module having at least one cameracapable of capturing frame images of a shopper within the retail store,said video capture module further comprising (i) an informationrecording unit capable of recording the captured frame images, and, (ii)an information storage unit capable of storing the captured frameimages; a video analytics module capable of extracting information aboutthe shopper state and behavior from the captured frame images, saidvideo analytics module further comprising (i) a captured frame imageclassification unit, said captured frame image classification unitoperable to perform selected processes on the captured frame imagesincluding selection recognition of shopper activity relative topurchasable items comprising predetermined behavior of physical handlingof the items, thereby generating a customer behavior class output, and,(ii) an information storage unit capable of storing the generatedcustomer behavior class output information; a planner module capable ofselecting the type of information to display on the digital signagebased on the extracted information from the video analytics module, saidplanner module further comprising (i) a captured frame image informationassociation unit wherein the captured frame images of shoppers capturedby the video capture module are compared to captured frame imagespreviously captured, and, digital signage capable of presenting theselected type of information from the planner module to the surroundingshoppers in the retail store.

According to aspects illustrated herein, there is provided a method forproviding information to shoppers from a display in a retail store basedon measured customer behaviors using a dynamic digital signage system,the method comprising: monitoring the display in the retail store bycapturing frame images of the display; identifying the presence of ashopper at the display in the retail store in the captured frame images;detecting the activity of the shopper at the display in the retail storein the captured frame images including selection recognition of shopperactivity relative to purchasable items comprising predetermined behaviorof physical handling of the items; processing the captured frame imagesto generate a shopper behavior class output; and, providing informationon the digital signage to the shopper at the display in the retail storebased on the customer behavior class output.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective illustration of a store utilizing a digitalsignage system in accordance with one embodiment of the presentdisclosure;

FIG. 2 is a functional block diagram illustrating a digital signagesystem in accordance with another embodiment of the present disclosure;

FIG. 3 is a flowchart illustrating a digital signage system inaccordance with another embodiment of the present disclosure; and,

FIG. 4 is a flowchart illustrating a digital signage system inaccordance with another embodiment of the present disclosure.

DETAILED DESCRIPTION

The present description and accompanying drawing FIGURES illustrate theembodiments of an improved digital signage system, and more particularlya dynamic digital signage system based on measured customer behaviorsthrough video analytics. Also provided is a more efficient method andsystem for providing useful information to shoppers in a retailenvironment using the presently described digital signage system.

FIG. 1 is a perspective illustration of a store 10 having a digitalsignage system in accordance with one embodiment of the presentdisclosure. As illustrated in FIG. 1, the store 10 is a retail store.Although the digital signage system is illustrated as being used withshoppers in a retail store, it can be appreciated that the digitalsignage system can be used with customers in other and/or alternativetypes of venues. A display 12 is positioned within the store 10 for thepurpose of displaying a variety of products and/or items such as, forexample, ties 14, shirts 16, etc. to one or more customers. The size ofthe display, shape of the display and products and/or items associatedwith the display 12 are non-limiting.

Signage 20 is illustrated as being positioned facing the front of thedisplay 12 such that as one or more customers approach the display 12,the signage 20 can be easily and conveniently viewed by the one or morecustomers. Signage 20 is also illustrated as being a digital signagesystem provided with an LCD display 22, and which is capable ofdisplaying selected types of information (e.g., marketing information,advertising information, promotional information, discount information,sale information, etc.) to the one or more customers at or near thedisplay 12. Signage 20 is also illustrated as being provided with acamera 24 for capturing frame images of a customer at or near the frontof the display 12. Camera 24 can be integrated into signage 20 and/orpositioned exterior to signage 20. Signage 20 is described in furtherdetail below.

With reference now to FIG. 2, there is illustrated a schematic diagramof a digital signage system 30 in accordance with another embodiment ofthe present disclosure. Digital signage system 30 is capable ofproviding dynamic adjustment of digital signage in a retail store basedon video-based measurements of shopper state and behavior in the retailenvironment. Digital signage system 30 is illustrated as including avideo capture module 32, a video analytics module 34, a planning module36 and digital signage 38.

Video capture module 32 is illustrated as including at least one camera40 designed to capture video in the form of frame images of one or moreshoppers within the retail store 10. In one embodiment, camera 40 isdesigned to capture frame images of shopper behavior within the retailstore 10. Camera 40 is typically positioned at or near a display 12 in astore 10, and positioned facing the front of display 12 so as to captureframe images of one or more customers as said one or more customersapproach the display 12. In the perspective illustration of FIG. 1, thecamera 24 of signage 20 corresponds to the camera 40 of video capturemodule 32.

Video capture module 32 can include an information recording unit 42 forrecording the captured frame images. Similarly, video capture module 34can include an information storage unit 44 for storing the capturedframe images. In one embodiment, the at least one camera 40 is asurveillance camera; however, the type of camera is non-limiting, andcan be any type of camera suitable for indoor and/or outdoor tracking ofone or more persons (e.g., camera, digital camera, video camera, digitalvideo camera, or combinations thereof, etc.). Depending on 1) the typeof camera used, and/or 2) the image capture geometry of the camera used,the frame rates of the camera can be in the range of from about 1 frameper second (fps) to about 300 fps, more typically from about 1.5 fps toabout 50 fps, and still more typically from about 2 fps to about 30 fps.For example, if camera 40 is a fisheye camera, a greater frame rate istypically used to accommodate non-linear pose and/or orientation effectsassociated with such a camera.

In another embodiment, the at least one camera 40 is used to trackshoppers within the field of view of the one or more cameras as theshoppers move throughout the store. Other and/or additional uses forcamera 40 can include providing an associated mapping from image capturecoordinates to real-world ground plane coordinates

The video analytics module 34 is provided downstream (i.e. positionedafter) from the one or more cameras 40 so as to receive information,such as, for example, captured frame images, from the one or morecameras 40. Video analytics module 34 is capable of overlayingpre-determined locations for displays, shelves, etc. within the field ofview of camera 40, thereby allowing a translation of the shoppercoordinates into semantic states such as, for example, “near the shirtand tie display”. The video analytics module 34 is also capable ofmeasuring the actions and/or behaviors of customers within the retailstore from the captured frame images including selection recognition ofshopper activity relative to purchasable items comprising predeterminedbehavior of physical handling of the items, and ultimately sending thisinformation to digital signage 38. As such, the digital signage systemcan present information to a shopper based on measured customerbehaviors of the physical handling of the purchasable item at a display.Given this information, the digital signage 38 in proximity to theshoppers can be adjusted so as to most effectively engage the shopper'sattention, and assist in their purchase decisions, thereby drivingcustomer satisfaction and sales. In one non-limiting example, the videoanalytics module 34 can identify that a particular shopper has picked upa shirt 16 at a display 12 but is actively searching for something thatgoes with it. Based on this information, a nearby digital signage 38 canoffer a suggestion for a matching item, such as, for example, a tie 14,in the same display and/or in a nearby display. In another non-limitingexample, the video analytics module 34 can identify that a particularshopper has been considering a particular item, such as, for example,shirt 16, for a long period of time. Based on this information, a nearbydigital signage 38 can offer a promotion (e.g., discount, two-for-one,etc.) if the item being considered is purchased.

The video analytics module 34 can include a captured frame imageclassification unit 46 operable to perform selected processes on thecaptured frame images including operations such as, for exampledetecting the behavior and/or activity of the shopper at the display inthe retail store in the captured frame images. For example, the videoanalytics module 34 is capable of using computer vision technology toconvert the captured frame images to actionable shopper behavior andstate information by operations including, but not limited to, objectdetection (e.g., customer detection, item detection, etc.) and/or objectidentification (e.g., specific customer identification, specific itemidentification, etc.). As such, the planner module 36 can use theclassifying output (i.e. the actionable shopper behavior and stateinformation) of the video analytics module 34 for displaying informationto the customer via digital signage 38. Video analytics module 34 isalso capable of using people tracking technology to provide informationregarding the location and/or position of the shoppers relative to keydisplays in the retail space. Additionally, video analytics module 34 iscapable of using image classification technology to classify key actionsand/or behaviors of interest, such as, for example, actions and/orbehaviors including picking up an item from a display, placing an itemin a basket and/or cart, placing the item back on the display, holdingthe item while evaluating the item, etc. Video analytics module 34 isalso capable of using image recognition technology to identify keyactions and/or behaviors of interest, such as, for example, picking upan item from a display, placing an item in a basket and/or cart, placingthe item back on the display, holding the item while evaluating theitem, etc. In addition, the video analytics module 34 is capable ofusing image decoding technology to decode a customer behavior and actionfrom the captured frame images. In one embodiment of the presentdisclosure, image-based classification on customer silhouettes is usedto distinguish simple actions such as, for example, picking an item,walking, browsing, etc. In another embodiment of the present disclosure,fitting a deformable parts model (DPM) to identify component elements ofhuman pose (e.g., arms, legs, torso, head, etc.) is used to detect thecustomer picking of an item from a display. Based on the captured frameimage classifying output of the video analytics module 34, a basic datarepresentation of where the customer is in the buying journey can bederived. As such, the video analytics module 34 is capable ofdetermining whether or not the customer is searching items, browsingitems, actively evaluating items, passively evaluating items, decidedagainst purchasing items (i.e. put item back on the display), decided topurchase items (i.e. put item in cart and/or basket, moving on withoutplacing item back on the display), etc. As such, the video analyticsmodule is capable of (i) monitoring the display in the retail store bycontinuously tracking dwell and location of the shopper by recognizingwhere the shopper is positioned in the captured frame images, (ii)determining whether the shopper is dwelling in front of the display inthe captured frame images, thereby identifying the presence of theshopper at the display in the retail store, (iii) classifying thebehavior of the shopper, thereby generating a shopper behavior classoutput including behaviors including picking up the purchasable itemfrom the display, placing the item in a basket or cart, placing the itemback on the display, or holding an item while evaluating the item, (iv)determining whether the shopper picked up the purchasable item from thedisplay in the captured frame images, (v) decoding the item picked up bythe shopper in the captured frame images, thereby generating an itemclass output including items including shirts, ties, pants, shorts,socks, or articles of clothing, (vi) determining whether the item countof items picked up by the shopper is greater than zero in the capturedframe images, and/or, (vii) determining whether the shopper put back anitem to the display in the captured frame images.

In one embodiment of the preset disclosure, video capture module 32 andvideo analytics module 34 of the digital signage system 30 can be usedto continuously track shoppers as they move throughout the store. Assuch, customer behavior from one region and/or department of the storecan be carried over to another region and/or department of the store.This information carried over to subsequent departments can assist theplanning module 36 in determining the most effective information fordisplay on the digital signage 38. In one embodiment, differentmessaging and/or promotions could be used at a second display, picked upbut put back items at the first display, or has picked up nothing fromthe first display. Path information (i.e. what order past displays werevisited by a shopper) can optionally be used in determining what shouldbe displayed on the digital signage 38.

The planning module 36 can include a captured frame image informationassociation unit 50 capable of making associations between situationalstates measured by the video analytics module 34 and desired informationto be displayed on the digital signage 38. Generally, the planningmodule 36 encodes domain and/or business knowledge of the most effectiveinformation (e.g., promotions, advertisements, pairings of clothing,etc.) for display on the digital signage 38 based on the currentbehaviors of surrounding shoppers. In one non-limiting embodiment, if ithas been determined by the video analytics module 32 that a shopper hasbeen evaluating an item for some length of time, it might be desirableto display one or more of the following on the digital signage 38:

-   -   a) a “buy-one-get-one” promotion that involves that item;    -   b) more information regarding the item of interest;    -   c) a glyph or other easily convertible link to enable the        shopper to use their mobile device to connect to more        information;    -   c) suggestions for other items that might pair well with the        item of interest; and/or,    -   d) for clothing items, a display of that item on a model with        other items that might pair well.

The relationships encoded within the planning module 36 can be based onsubject matter experts in the retail space, on data mining and/ormachine learning of past sales, designed experiments specificallyintended to correlate sales and key factors (e.g., merchandising,promotions, etc.), or combinations thereof, etc.

Digital signage 38 is typically positioned at or near a display 12 in astore 10 so as to provide easy and convenient viewing for one or moreshoppers in store 10. In the perspective illustration of FIG. 1, thesignage 20 corresponds to the digital signage 38 of digital signagesystem 30.

In one embodiment, the digital signage system 30 can be used to provideinformation to shoppers at or near the display 12 in the retail store10, wherein the information provided can be based on measured customerbehaviors. As such, the digital signage system 30 can 1) monitor thedisplay 12 in the retail store 10 by capturing frame images of thedisplay, 2) identify the presence of a shopper at the display 12 inretail store 10 in the captured frame images, 3) detect the activity ofthe shopper at the display 12 in retail store 10 the captured frameimages, 4) process the captured frame images to generate a customerbehavior class output, and/or 5) provide information to the shopper ondigital signage 38 at the display 12 in retail store 10 based on thecustomer behavior class output.

A flowchart of the basic state and/or behavior estimation for a shopperis illustrated in FIG. 3.

In operation, camera 40 can continuously monitor the front of a display12 positioned in a store 10, thereby providing a steady flow of video inthe form of captured frame images to video analytics module 34.Additionally, the time at which the captured frame image was capturedand/or the camera identification information of the camera whichcaptured the captured frame images can be sent to the video analyticsmodule 34. The video analytics module 34 can store one or more of 1) thecaptured frame image, 2) the time the captured frame image was captured,and/or 3) the camera identification information of the camera from whichthe captured frame image was captured in an information storage unit 48.Video analytics module 34 initiates the process (neutral) as shown inFIG. 3 upon receiving captured frame images from video capture module32. During the process (neutral), the digital signage 38 can providebasic marketing information, advertising information, sale information,promotional information, etc. of items sold in the store 10.

In operation, camera 40 of video capture module 32 continuously capturesframe images of customers as the customers walk throughout the store 10and simultaneously sends the captured frame images to the videoanalytics module 34. In operation, as a customer approaches the display,the video analytics module 34 identifies the presence of the shopper ator near the display 12 by recognizing, such as, for example, by usingtrained object detection classifiers as are common in the field ofcomputer vision, whether or not a customer is present in front of thedisplay 12 in the captured frame images. As such, the video analyticsmodule 34 can continuously track the dwell and location of the shopper(step S1) by leveraging computer vision methods, such as, for example,by tracking appearance features like color, color histogram, histogramof oriented gradients (HOG) or alternative differentiating appearancefeatures on each customer, where the customer is positioned in thecaptured frame images. Similarly, the video analytics module 34determines, such as, for example, by tracking individual customertrajectories, whether or not the shopper is dwelling in front of adisplay X (step S2) in the captured frame images.

When it is determined that a customer is not dwelling in front ofdisplay X in the captured frame images (step S2; No), video analyticsmodule 34 concludes the processing of the captured frame images, andcontinues to track the dwell and location of the customer in thecaptured frame images (step S1). As illustrated in FIG. 3, thesuccessive recognition of “track shopper location and dwell” (step S1)and “dwelling in front of display X?” (step S2) actions can optionallybe continued until it is determined that a customer is dwelling in frontof a display X in the captured frame images (step S2; Yes).

When it is determined that a customer is dwelling in front of display Xin the captured frame image (step S2; Yes), the video analytics module34 signals the planner module 36 that the customer is browsing thedisplay (step S3), thereby initiating the process (browsing). During theprocess (browsing), digital signage 38 typically provides more specificmarketing information, advertising information, sale information,promotional information, etc. for items sold at display 12 in store 10.

Similar to the process (neutral), during the process (browsing), camera40 of video capture module 32 continuously captures frame images fromvideo taken of the customer at the display, and sends the captured frameimages to the video analytics module 34. As such, the video analyticsmodule 34 can continuously track the dwell and location of the shopper(step S4) by leveraging computer vision methods, such as, for example,by tracking appearance features like color, color histogram, histogramof oriented gradients (HOG) or alternative differentiating appearancefeatures on each customer, where the customer is positioned in thecaptured frame images. Similarly, the video analytics module 34determines whether or not the shopper is dwelling in front of display Xin the captured frame images (step S5).

When it is determined that a customer is not dwelling in front ofdisplay X in the captured frame images (step S5; No), video analyticsmodule 34 concludes the process (browsing) and initiates the process(leaving). In one embodiment, during the process (leaving), the specificinformation from the process (browsing) is removed from digital signage38, and the more basic information from the process (neutral) istypically displayed on digital signage 38. In another embodiment, duringthe process (leaving), the specific information displayed on the digitalsignage 38 during the process (browsing) can be exemplified such as, forexample, by flashing, scrolling, etc. to attract the customers'attention. After initiating the process (leaving), the video analyticsmodule 34 signals the planner module 36 that the shopper is leaving thedisplay (step S6), thereby initiating the process (neutral).

When it is determined that a customer is dwelling in front of display Xin the captured frame images (step S5; Yes), the video analytics module34 can optionally classify, such as, for example, by bag-of-featuresbased classification methods using HOG3D or dense trajectory features,the action and/or activity of the shopper in front of display X in thecaptured frame images (step S7), thereby generating a customer behaviorclass output. The customer behavior class output can include activitiesand/or behaviors including, but not limited to, picking up an item froma display, placing an item in a basket and/or cart, placing an item backon a display, holding an item while evaluating the item, etc.Optionally, the customer action and/or activity can be recorded andstored in the same information storage unit as 1) the captured frameimage, 2) the time the captured frame image was captured, and/or 3) thecamera identification information of the camera from which the capturedframe image was captured. After optionally classifying the action and/oractivity of the customer, and recording and/or storing the action and/oractivity of the customer in the information storage unit, the videoanalytics module 34 can determine, such as, for example, bybag-of-features based classification methods using HOG3D or densetrajectory features, whether or not the shopper picked up an item fromthe display in the captured frame images (step S8).

When it is determined that the customer has not picked up an item in thecaptured frame image (step S8; No), the video analytics module 34continues the (browsing) process, and continues to track the dwell andlocation of customers in the captured frame images (step S4). Asillustrated in FIG. 3, the successive recognition of “dwelling in frontof display X?” (step S5) and “picked up an item?” (step S8) actions canoptionally be continued until it is determined that a customer haspicked up an item in the captured frame images (step S8; Yes).

When it is determined that a customer has picked up an item in thecaptured frame images (step S8; Yes), the video analytics module 34concludes the process (browsing) and initiates the process(considering). At this point in the customers' browsing journey, theshopper has picked up an item from the display and is consideringpurchasing the item. As such, during the process (considering), digitalsignage 38 typically provides specific marketing information,advertising information, sale information, promotional information, etc.for the item picked by the customer at display 12 in store 10.Additionally, when it is determined that a customer has picked up anitem in the captured frame images (step S8; Yes), the view analyticsmodule is capable of extracting information about the shopper behaviorand state from the captured frame images including selection recognitionof shopper activity relative to purchasable items comprisingpredetermined behavior of physical handling of the items.

After initiating the process (considering), the video analytics module34 can optionally decode, such as, for example, by common computervision methods for object detection and recognition including leveragingbag-of-features based image classification on scale-invariant featuretransform (SIFT), histogram of oriented gradients (HOG), or convolutionneural-network (CNN) based features, the item picked up by the shopperin the captured frame images (step S9), thereby generating an item classoutput. The item class output can include items including, but notlimited to, shirts, ties, pants, shorts, socks, etc. Optionally, theitem decoded from the captured frame images can be recorded and storedin the same information storage unit as 1) the captured frame image, 2)the time the captured frame image was captured, 3) the cameraidentification information of the camera from which the captured frameimage was captured, and/or 4) the customer action and/or activity. Afteroptionally decoding the item picked up by the shopper (step S9),recording the item picked up by the shopper, and/or storing the itempicked up by the shopper, the video analytics module 34 can signal theplanner module 36 1) the customer picked up an item, and/or 2) the typeof item the customer picked up (step S10), thereby initiating theprocess (browsing).

Referring now to FIG. 4, there is provided a flowchart illustratingother and/or additional steps between the processes (considering) and(browsing) of FIG. 3.

When it is determined, such as, for example, by bag-of-features basedclassification methods using HOG3D or dense trajectory features, that acustomer has picked up an item in the captured frame images (step S8;Yes), the video analytics module 34 initiates the process (considering).After initiating the process (considering), the video analytics module34 can set the item count at one item (step S11), the one itemrepresenting the item picked up by the customer (step S8; Yes). Thevideo analytics module 34 can then optionally decode, such as, forexample, by common computer vision methods for object detection andrecognition including leveraging bag-of-features based imageclassification on scale-invariant feature transform (SIFT), histogram oforiented gradients (HOG), or convolution neural-network (CNN) basedfeatures, the item picked up by the shopper in the captured frame images(step S12), thereby generating an item class output. The item classoutput can include items including, but not limited to, shirts, ties,pants, shorts, socks etc. Optionally, the item decoded from the capturedframe images can be recorded and stored in the same information storageunit as 1) the captured frame image, 2) the time the captured frameimage was captured, 3) the camera identification information of thecamera from which the captured frame image was captured, and/or 4) thecustomer action and/or activity.

After optionally decoding the item picked up by the shopper (step S12),recording the item picked up by the shopper, and/or storing the itempicked up by the shopper in an information storage unit 48, the videoanalytics module 34 can signal the planner module 36 1) the customerpicked up an item, 2) the type of item the customer picked up, 3) theaction and/or activity of the customer, 4) the display from which thecustomer picked up an item, etc. (step S13).

From the captured frame images, the video analytics module 34 candetermine, such as, for example, by leveraging computer vision methodsto identify the “pick up” and “put back” actions, whether or not theitem count of items picked up by the shopper is greater than zero (stepS14). When it is determined that the item count is not greater than zero(i.e. no item has been picked up, an item was returned to the display,etc.), as illustrated by (step S14; No), the video analytics moduleconcludes the process (considering) and initiates the process(browsing). During the transition from process (considering) to process(browsing), the item-specific information from the process (considering)is removed from the digital signage 38, and more general iteminformation from the process (browsing) is typically presented ondigital signage 38. Additionally, during the process (browsing), asdescribed earlier, camera 40 of video capture module 32 continuouslycaptures frame images from video taken of the customer at the display,and sends the captured frame images to the video analytics module 34 forfurther processing by image classification unit 46 of video analyticsmodule 34.

When it is determined that the item count is greater than zero in thecaptured frame images (step S14; Yes), the video analytics module 34continues to track the dwell and location of the shopper (step S15) bymethods previously described. After it has been determined that the itemcount is greater than zero (step S14; Yes), the video analytics module34 can determine, such as, for example, by trajectory analytics, whetheror not the customer is still browsing display X in the captured frameimages (step S16).

When it is determined that the customer is not still browsing display Xin the captured frame images (step S16; No), video analytics moduledetermines, such as, for example, by leveraging computer vision methodsto identify the “pick up” and “put back” actions, whether or not theitem count of items picked up by the shopper is greater than zero (stepS17). When it is determined that the item count is not greater than zero(i.e. no item has been picked up, an item was returned to the display,etc.) in the captured frame images (step S17; No), the video analyticsmodule 34 can signal the planner module 36 a pending lost opportunityfor sale (step S18), thereby initiating the process (neutral).Similarly, when it is determined that the item count is greater thanzero (step S17; Yes), the video analytics module 34 can signal theplanner module 36 (step S19) a pending sale, thereby initiating theprocess (neutral). During the transition to the process (neutral), theitem-specific information from the process (considering) can be removedfrom the digital signage 38, and basic information, such as, forexample, marketing information, advertising information, saleinformation, promotional information, etc. can typically be presented ondigital signage 38.

When it is determined that a customer is still browsing display X in thecaptured frame images (step S16; Yes), the video analytics module 34 canoptionally classify, such as, for example, by a bag-of-features basedclassification methods using HOG3D or dense trajectory features on aninput image, the action and/or activity of the shopper in front ofdisplay X in the captured frame images (step S20), thereby generating acustomer behavior class output. The customer behavior class output caninclude activities and/or behaviors including, but not limited to,picking up an item from a display, placing an item in a basket and/orcart, placing an item back on a display, holding an item whileevaluating the item, etc. Optionally, the customer action and/oractivity can be recorded and stored in the same information storage unitas 1) the captured frame image, 2) the time the captured frame image wascaptured, and/or 3) the camera identification information of the camerafrom which the captured frame image was captured. Optionally, in retailenvironments where a plurality of digital signage is used, leveraginginformation based on the customer's action and/or activity recorded atone display can affect and/or influence the digital signage content at asubsequent or alternative retail display, such as, for example, digitalsignage positioned at an additional location within the retail store.After classifying the action and/or activity of the customer (step S20),recording the action and/or activity of the customer, and/or storing theaction and/or activity of the customer in the information storage unit48, the video analytics module 34 can determine, such as, for example,by common computer vision methods for object detection and recognitionincluding leveraging bag-of-features based image classification onscale-invariant feature transform (SIFT), histogram of orientedgradients (HOG), or convolution neural-network (CNN) based features,whether or not the shopper put back an item to the display in thecaptured frame images (step S21).

When it is determined that the customer did put back an item in thecapture frame images (step S21; Yes), the video analytics module 34 candecrement the item count (step S24). After the item count is decremented(step S24), the video analytics module 34 can decode, such as, forexample, by a computer programmed to perform decoding of an input image,item returned to the display by the customer, and/or the item still heldby the customer in the captured frame images (step S12).

When it is determined that the customer did not put back an item in thecaptured frame images (step S21; Yes), the video analytics module 34 candetermine, such as, for example, by a bag-of-features basedclassification methods using HOG3D or dense trajectory features on aninput image, whether or not the customer picked up another item in thecaptured frame images (step S22). When it is determined from thecaptured frame images that the customer did not pick up another item(step S22; No), the item held by the customer can be decoded in thecaptured frame images (step S12) such as, for example, by a computerprogrammed to perform decoding of an input image. When it is determinedfrom the captured frame images that the customer did pick up anotheritem in the captured frame images (step S22; Yes), the video analyticsmodule 34 can increment the item count (step S23). After the item countis incremented (step S23), the video analytics module 34 can decode,such as, for example, by a computer programmed to perform decoding of aninput image, the item held by the customer, and/or the additional itempicked up by the customer (step S12).

As illustrated in FIG. 4, the number of items being considered by theshopper is determined by successive recognition of the “pick up” (stepS22) and “put back” (step S21) actions. This additional informationcould be useful for the planning module 36 in determining what the mostappropriate information would be to display on the digital signage 38.In one embodiment, if the shopper has picked up more than one item, itmight be desirable to provide information to the customer on“buy-on-get-one” type promotions involving the items at that display. Inanother embodiment, if the shopper has only picked up a single item, itmight be desirable to offer suggestions on the digital signage 38 forother items of interest at the display or in other areas of the store.

It will be appreciated that the variants of the above-disclosed andother features and functions, or alternatives thereof, may be combinedinto many other different systems or applications. Various presentlyunforeseen or unanticipated alternatives, modifications, variations orimprovements therein may be subsequently made by those skilled in theart which are also intended to be encompassed by the following claims.

What is claimed is:
 1. A system capable of providing dynamic adjustmentof digital signage in a retail store based on video-based measurementsof shopper state and behavior in the surrounding scene, the systemcomprising: a video capture module having at least one camera capable ofcapturing frame images of a shopper within the retail store; a videoanalytics module capable of extracting information about the shopper'sbehavior and state from the captured frame images including selectionrecognition of shopper activity relative to purchasable items comprisingpredetermined behavior of physical handling of the items; a plannermodule capable of selecting a type of information to display on thedigital signage based on the extracted information from the videoanalytics module; and, digital signage capable of presenting theselected type of information from the planner module to surroundingshoppers in the retail store.
 2. The system of claim 1 wherein the videocapture module further comprises: (i) an information recording unitcapable of recording the captured frame images; and, (ii) an informationstorage unit capable of storing the captured frame images.
 3. The systemof claim 1 wherein the camera is selected from the group consisting of adigital camera, a video camera, or digital video camera.
 4. The systemof claim 1 wherein the video analytics module further comprises: (i) acaptured frame image classification unit, said captured frame imageclassification unit operable to perform selected processes on thecaptured frame images, thereby generating a shopper behavior classoutput; and, (ii) an information storage unit capable of storinginformation including the generated shopper behavior class outputinformation, the captured frame image, the time the captured frame imagewas captured, the camera identification information of the camera fromwhich the captured frame image was captured, the shopper behavior and/oractivity, and the item decoded from the captured frame images.
 5. Thesystem of claim 4 wherein the captured frame image classification unitis capable of processing the captured frame images by operationsincluding the use of computer vision technology, people trackingtechnology, image classification technology, image recognitiontechnology, or image decoding technology.
 6. The system of claim 1wherein the planner module further comprises: (i) a captured frame imageinformation association unit wherein the captured frame images ofshoppers captured by the video capture module are compared to capturedframe images previously captured.
 7. A system capable of providingdynamic adjustment of digital signage in a retail store based onvideo-based measurements of shopper state and behavior, the systemcomprising: a video capture module having at least one camera capable ofcapturing frame images of a shopper within the retail store, said videocapture module further comprising: (i) an information recording unitcapable of recording the captured frame images; and, (ii) an informationstorage unit capable of storing the captured frame images; a videoanalytics module capable of extracting information about the shopperstate and behavior from the captured frame images, said video analyticsmodule further comprising: (i) a captured frame image classificationunit, said captured frame image classification unit operable to performselected processes on the captured frame images including selectionrecognition of shopper activity relative to purchasable items comprisingpredetermined behavior of physical handling of the items, therebygenerating a shopper behavior class output; and, (ii) an informationstorage unit capable of storing the generated shopper behavior classoutput information; a planner module capable of selecting the type ofinformation to display on the digital signage based on the extractedinformation from the video analytics module, said planner module furthercomprising: (i) a captured frame image information association unitwherein the captured frame images of shoppers captured by the videocapture module are compared to captured frame images previouslycaptured; and, digital signage capable of presenting the selected typeof information from the planner module to the surrounding shoppers inthe retail store.
 8. The system of claim 7 wherein the captured frameimage classification unit is capable of processing the captured frameimages by operations including the use of computer vision technology,people tracking technology, image classification technology, imagerecognition technology, or image decoding technology.
 9. The system ofclaim 7 wherein the information storage unit is capable of storinginformation including the captured frame image, the time the capturedframe image was captured, the camera identification information of thecamera from which the captured frame image was captured, the shopperaction and/or activity, or the item decoded from the captured frameimages.
 10. A method for providing information to shoppers from adisplay in a retail store based on measured shopper behaviors using adynamic digital signage system, the method comprising: monitoring thedisplay in the retail store by capturing frame images of the display;identifying the presence of a shopper at the display in the retail storein the captured frame images; detecting the activity of the shopper atthe display in the retail store in the captured frame images includingselection recognition of shopper activity relative to purchasable itemscomprising predetermined behavior of physical handling of the items;processing the captured frame images to generate a shopper behaviorclass output; and, providing information on the digital signage to theshopper at the display in the retail store based on the shopper behaviorclass output.
 11. The method of claim 10 wherein the digital signagesystem comprises: a video capture module having at least one cameracapable of capturing frame images of a shopper within the retail store;a video analytics module capable of extracting information about theshopper's behavior and state from the captured frame images includingselection recognition of shopper activity relative to purchasable itemscomprising predetermined behavior of physical handling of the items; aplanner module capable of selecting the type of information to displayon the digital signage based on the extracted information from the videoanalytics module; and, digital signage capable of presenting theselected type of information from the planner module to shoppers in theretail store, wherein leveraging information measured and recorded bythe digital signage system at a first display can affect digital signagecontent at a second display.
 12. The method of claim 11 wherein thevideo analytics module is further capable of: (i) monitoring the displayin the retail store by continuously tracking dwell and location of theshopper by recognizing where the shopper is positioned in the capturedframe images; (ii) determining whether the shopper is dwelling in frontof the display in the captured frame images, thereby identifying thepresence of the shopper at the display in the retail store; (iii)classifying the behavior of the shopper, thereby generating a shopperbehavior class output including behaviors including picking up thepurchasable item from the display, placing the item in a basket or cart,placing the item back on the display, or holding an item whileevaluating the item; (iv) determining whether the shopper picked up thepurchasable item from the display in the captured frame images; (v)decoding the item picked up by the shopper in the captured frame images,thereby generating an item class output including items includingshirts, ties, pants, shorts, socks, or articles of clothing; (vi)determining whether the item count of items picked up by the shopper isgreater than zero in the captured frame images; and, (vii) determiningwhether the shopper put back an item to the display in the capturedframe images.
 13. The method of claim 12 wherein upon determination thatthe shopper is not dwelling at the display in the captured frame images,the video analytics module tracks dwell and location of the shopper inthe captured frame images, and upon determination that the shopper isdwelling at the display in the captured frame images, the videoanalytics module signals the planner module that the shopper is browsingthe display.
 14. The method of claim 12 wherein upon determination thatthe shopper did not pick up the item from the display, the videoanalytics module tracks dwell and location of the shopper in thecaptured frame images, and upon determination that the shopper picked upthe item, the video analytics module signals the planner module that theshopper is considering purchasing the item picked up at display.
 15. Themethod of claim 12 wherein upon determination that the item count is notgreater than zero, the video analytics module signals the planner modulethat the shopper is browsing purchasable items at the display, and upondetermination that the item count is greater than zero, the videoanalytics module tracks dwell and location of the shopper in thecaptured frame images.
 16. The method of claim 15 wherein upondetermination that the item count of purchasable items picked up by theshopper is greater than zero in the captured frame images, the videoanalytics module determines whether the shopper is still browsing thedisplay.
 17. The method of claim 16 wherein upon determination that theshopper is not still browsing the display, the video analytics moduledetermines whether the item count is greater than zero, and upondetermination that the shopper is still browsing the display, the videoanalytics module classifies the shopper behavior, thereby generating ashopper behavior class output including behaviors including picking upan item from the display, placing an item in a basket or cart, placingan item back on the display, or holding an item while evaluating theitem.
 18. The method of claim 12 wherein upon determination that theitem count is not greater than zero, the video analytics module signalsthe planner module a pending lost opportunity for sale, and upondetermination that the item count is greater than zero, the videoanalytics module signals the planner module a pending sale.
 19. Themethod of claim 12 wherein upon determination that the shopper did putback an item in the captured frame images, the video analytics moduledecrements the item count, and upon determination that the shopper didnot put back an item in the captured frame images, the video analyticsmodule determines whether the shopper picked up another item from thedisplay in the captured frame images.
 20. The method of claim 12 whereinupon determination that the shopper did not pick up another item fromthe display, the video analytics module decodes the item held by theshopper, and upon determination that the shopper did pick up anotheritem, the video analytics module increments the item count.